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ABSTRACT 


The  advent  of  Light  Detection  and  Ranging  (LiDAR)  point  cloud  collection  has 
significantly  improved  the  ability  to  model  the  world  in  precise,  fine,  three-dimensional 
details.  The  objective  of  this  research  was  to  demonstrate  accurate,  foundation  methods 
for  fusing  LiDAR  data  and  photogrammetric  imagery  and  their  potential  for  change 
detection.  The  scope  of  the  project  was  to  investigate  optical  image  to  LiDAR 
registration  methods,  focusing  on  several  dissimilar  image  types  including  Optical  Bar 
Camera  (OBC),  high  resolution  aerial  frame,  and  WorldView  1  satellite  with  varying 
LiDAR  point  densities.  An  innovative  optical  image  to  LiDAR  data  registration  process 
was  established.  This  approach  was  demonstrated  for  one  image  type  using  the  rational 
polynomial  coefficients  (RPC)  representation  of  the  panoramic  math  model  improving 
accuracy  from  1.9  m  to  0.5  m  root  mean  square  (RMS)  error.  Comparison  of  stereo 
imagery  point  cloud  data  to  the  LiDAR  point  cloud  using  a  90%  confidence  interval 
highlighted  changes  that  included  small  scale  (<  50cm),  sensor  dependent  change  and 
large  scale,  new  home  construction  change.  This  research  also  proposed  a  fused  LiDAR 
and  stereo  image  base  layer  as  the  foundation  for  further  LiDAR/image  fusion. 
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I.  INTRODUCTION 


Light  Detection  and  Ranging  (LiDAR)  uses  the  measurement  of  timed  light 
pulses  to  determine  elevation  differences.  It  produces  “point  clouds”  of  3D  data  of 
varying  density  depending  on  the  light  source  and  interaction  with  the  Earth’s  surface. 
LiDAR  point  cloud  data  have  an  outstanding  ability  to  represent  and  portray  topography, 
building  structures,  and  foliage  with  precise,  fine,  three  dimensional  details.  (Shan  & 
Toth,  2009)  The  precision  required  for  Light  Detection  and  Ranging  (LiDAR)  point 
cloud  data  to  successfully  model  the  earth’s  surface  and  features  is  an  open  question.  The 
goals  of  LiDAR  measurements  are  similar  to  those  of  Photogrammetry.  In 
Photogrammetry,  the  objective  is  to  extract  precision  three-dimensional  representations 
of  the  earth’s  surface  using  analysis  of  parallax  in  stereo  images.  Even  when  viewed  as 
photographs,  photogrammetric  images  can  appear  precise  because  of  the  many  fine  and 
structured  details  that  are  visible  and  interpreted.  LiDAR  uses  direct  measurement  of 
distances  with  respect  to  known  ground  control  to  determine  point  clouds  that  can  be 
further  analyzed  to  generate  Digital  Surface  Models  (DSM)  or  Digital  Elevation  Models 
(DEM). The  nature  of  these  two  data  sets  makes  them  ideal  for  improved  3D 
representation  of  surfaces.  To  successfully  accomplish  this,  however,  the  precision  and 
inherent  errors  of  each  dataset  need  to  be  well  understood. 

It  is  common  for  LiDAR  point  cloud  data  and  Photogrammetric  quality  imagery 
(mono  and  stereo)  to  be  collected  at  different  times  over  an  area  of  interest.  This 
difference  in  collection  times  can  span  several  months  to  years.  This  fact  makes  co¬ 
registration  and  fusion  challenging. 

This  research  demonstrates  the  processes  and  results  of  fusing  LiDAR  point  cloud 
data  with  photogrammetric  imagery  and  derived  stereo  photogrammetric  image  point 
cloud  data.  The  process  highlights  innovative  techniques  of  registering  Optical  imagery 
to  LiDAR  data.  In  addition,  the  process  demonstrates  the  improved  registration  accuracy 
when  Rational  Polynomial  Coefficients  (RPC)  support  data  are  used  in  place  of  the 
rigorous  panoramic  model  for  the  Optical  Bar  Camera  (OBC).  Once  the  stereo  optical 

imagery  are  coincident  with  the  LiDAR  data,  fusion  of  optical  imagery  with  LiDAR  point 

1 


cloud  data  and  fusion  of  stereo  image  derived  point  cloud  data  and  LiDAR  point  cloud 
data  can  be  accomplished. 

We  usually  think  about  data  fusion  as  a  resource  to  answer  a  specific  question  or 
as  a  tool  to  tackle  a  specific  problem.  Data  fusion  can  also  be  a  means  to  create  a  new 
data  set  that  is  more  readily  exploitable.  Several  examples  of  using  data  fusion  to  create 
a  more  useable  dataset  exist.  For  example,  the  fusion  of  multispectral  and  panchromatic 
imagery  is  used  to  create  a  “pan-sharpened”  or  “resolution  merge”  image  (Pohl  &  Van 
Genderen,  1998).  Usually  the  multispectral  image  is  lower  resolution  than  the 
panchromatic  image  and  the  goal  is  to  create  a  true  color  image  from  the  multispectral 
image  and  then  increase  the  resolution  of  this  true  color  image  by  using  the  panchromatic 
image  in  a  sub-pixel  sampling  process.  This  process  has  become  so  common  that  we 
often  do  not  think  about  it  as  a  data  fusion  process.  Pan-sharpening  imagery  is  an 
elegant  example  of  how  fusion  of  data  sets  can  make  a  new  data  set  that  is  more  useable 
and  ready  for  exploitation. 

Taking  the  above  example  a  step  further,  elevation  data  can  used  to  create  an 
orthophoto  of  the  pan-sharpened  image.  This  three  source  data  fusion  process  of  ortho- 
rectifying  pan-sharpened  color  imagery  creates  a  base  layer  of  imagery  that  can  be  map 
quality  and  useable  for  precise  feature  extraction.  We  exploit  the  results  of  this  three 
source  fusion  process  every  day  when  we  open  Google  Earth™  or  Yahoo  Maps™. 

The  purpose  of  these  examples  was  to  show  how  geospatial  data  can  be  fused  to 
create  more  useable  and  exploitable  data  sets.  This  fusion  process  can  extend  into  other 
remote  sensing  phenomenologies  such  as  LiDAR  and  Hyperspectral  Imagery,  Radar  and 
Optical  Imagery,  and  LiDAR  and  Optical  imagery.  The  focus  of  this  research  was  to 
study  the  results  of  fusing  LiDAR  point  cloud  data  with  Optical  imagery  and  Stereo 
Optical  imagery  derived  point  cloud  data. 
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II.  BACKGROUND 


A.  HISTORY  OF  THE  PROBLEM 

LiDAR  point  cloud  data  can  contain  a  lot  of  information  in  3D  space.  If  dense 
enough,  this  can  be  very  useful  at  the  individual  feature  level  and  can  provide  realistic 
views  that  can  be  rotated  and  viewed  from  many  angles  (Figure  1). 


MSL  Height 
■■  -  36  m 


I  -  25  m 

-  14  m 


Scale: 


15  m 


30  m 


Location:  36.596  N  121.874  W 


Figure  1 .  LiDAR  point  cloud  data  view  showing  individual  3D  feature  level 
information.  Individual  points  are  color  coded  by 
Mean  Sea  Level  (MSL)  height. 


The  application  of  laser  ranging  and  point  cloud  collection  has  exploded  in  the 
last  decade.  Close  range  modeling,  terrestrial  navigation,  and  airborne  topography  are 
varied  examples  of  problems  that  continue  to  drive  a  need  for  precision  LiDAR  point 
cloud  data.  The  fundamentals  of  using  a  laser  to  collect  ranging  or  distance  information 
in  large  volume  apply  to  many  3D  knowledge  problems. 
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The  history  of  LiDAR  and  photogrammetry  fusion  shows  that  many  individuals 
with  diverse  interests  are  studying  this  problem.  Experts  from  the  Computer  Vision 
community,  Photogrammetry  community,  and  LiDAR  community  all  have  an  interest  in 
solving  the  problems  of  LiDAR  and  imagery  fusion  (Kaminsky  et  ah,  2009;  Schenk  et  ah, 
2001;  Rodarmel  et  ah,  2006).  After  comprehensive  study,  these  diverse  disciplines  show 
significant  overlap  and  the  problems  to  be  solved  are  similar.  No  matter  which  one  of 
these  disciplines  chosen,  rapid  collection  of  precision  point  cloud  data  and  extraction  of 
the  3D  information  content  of  that  data  are  the  goals. 

Registering  LiDAR  collected  point  cloud  data  to  optical  imagery  is  not  as  easy  as 
it  sounds  because  point  cloud  data  contains  discrete  points  that  are  not  easy  to  precisely 
locate  on  an  optical  image.  Lor  instance,  on  an  optical  image  with  the  proper  resolution, 
the  comer  of  a  sidewalk  or  building  or  roof  top  can  be  precisely  identified  and  measured. 
With  LiDAR  point  cloud  data,  the  exact  comer  of  a  building  usually  does  not  contain  a 
collected  point.  This  presents  a  dilemma  for  the  registration  of  these  two  data  sets. 

Two  prevailing  methods  have  been  developed  to  overcome  this  registration 
dilemma  (Mishra  &  Zhang,  2012).  The  first  method  uses  derived  edges  from  both  data 
sets  and  attempts  to  match  them  up.  The  second  method  uses  the  LiDAR  derived 
intensity  image  and  DEM  data  to  control  the  optical  image. 

Computer  Vision  solutions  to  this  fusion  problem  have  grown  in  popularity. 
Many  reports  elaborate  on  how  computer  vision  multi-image  matching  has  stepped  up  in 
place  of  photogrammetry  to  build  point  clouds  (Leberl  et  ah,  2010).  One  Computer 
Vision  approach  to  the  problem  is  to  build  the  point  cloud  from  numerous  overlapping 
images  in  an  arbitrary  image  model  space  and  then  align  this  point  cloud  to  imagery 
through  classic  point,  edge,  and  homogeneous  area  matching  (Kaminsky  et  al.,  2009). 

While  Computer  Vision  solutions  are  new  and  attractive,  they  rely  heavily  on 
image  content  such  as  building  structures  and  roads  that  have  strong  edges  and  features  to 
perform  image  and  point  cloud  matching  .  The  idea  of  using  linear  features  for  matching 
is  also  prevalent  in  the  photogrammetry  community  as  it  has  been  successfully 
implemented  in  image  to  image  matching  routines  (Habib,  Ghanma,  &  Kim,  2005).  A 
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quick  visualization  of  an  optical  image  and  LiDAR  point  cloud  shows  the  strong  edges  of 
both  data  sets  that  would  make  ideal  matching  features  (Figure  2). 
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Figure  2.  Side-by-side  view  of  a  high  resolution  color  frame  image  (left) 

and  top  view  of  a  LiDAR  point  cloud  (right)  highlighting  linear  edge 
features  ideal  for  automatic  registration. 


Areas  where  topography  is  being  derived,  unfortunately,  do  not  always  contain 
urban  structure,  and  tend  to  be  very  homogeneous  in  image  content.  As  a  result, 
topographic  and  mapping  applications  tend  to  lean  toward  using  the  intensity  image  and 
LiDAR  derived  Digital  Elevation  Model  (DEM)  as  a  reference  base.  This  solution  for 
registration  is  not  popular,  however,  because  it  is  not  well  suited  for  automation  (Mishra 
&  Zhang,  2012). 

One  registration  approach  encountered  is  to  create  two  images  from  the  LiDAR 
data  (Habib,  Ghanma,  &  Kim,  2005).  One  image  is  the  traditional  intensity  image  and 
the  second  image  is  the  DEM  represented  as  a  “range”  image.  This  does  not  seem  very 
accurate  though  because  the  height  value  is  fixed  for  any  given  sub-pixel  location.  A 
better  approach  would  be  to  interpolate  a  height  value  between  the  DEM  points. 

Many  traditional  photogrammetry  researchers  have  recognized  the  mutual 

benefits  that  a  photogrammetric  derived  point  cloud  and  a  LiDAR  point  cloud  can 
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provide  (White  et  al.,  2013;  Habib  et  al,  2008).  The  problem  of  fusing  LiDAR  data  and 
photogrammetric  imagery  becomes  much  more  complex  when  registering  large-scale 
imagery  and  attempting  to  create  true  orthophotos  (Habib  et  al,  2008).  True  orthophotos 
are  different  than  a  terrain  corrected  orthophoto.  True  orthophotos  use  detailed  3D 
feature  information  to  stand  buildings  up.  In  the  typical  terrain  corrected  orthophoto, 
buildings  and  tall  features  will  still  retain  the  perspective  view  from  the  photogrammetric 
image  often  showing  lean  away  from  the  center  of  the  image.  The  Digital  Surface  Model 
(DSM)  can  be  more  completely  described  with  the  fusion  of  the  LiDAR  point  cloud  and 
stereo  point  cloud,  resulting  in  better  representation  of  the  3D  environment.  This 
improved  representation  of  the  3D  environment  can  produce  more  precise  true 
orthophotos. 

Outside  of  the  urban  environment,  forest  management  has  started  to  engage  the 
utility  of  LiDAR  point  cloud  data  and  how  it  can  be  used  with  the  more  traditional  aerial 
photogrammetric  practices  today  (White  et  al.,  2013).  The  partial  tree  canopy  penetration 
capabilities  of  LiDAR  help  to  establish  tree  height  and  tree  maturity.  Also,  the 
underlying  Digital  Terrain  Model  (DTM)  can  be  created  from  the  LiDAR  data  to 
understand  water  drainage  and  road  access.  These  data  can  then  be  fused  with 
photogrammetric  and  multispectral  image  derived  tree  species  and  tree  health 
information,  resulting  in  a  complete  model  of  the  forest  inventory. 

LiDAR  collected  point  clouds  and  stereo-image-derived  point  clouds  can  also  be 
used  for  change  detection  (Schenk  &  Csatho,  2003),  which  is  one  of  the  focuses  of  this 
research.  The  derived  data  from  each  sensor  can  be  used  to  investigate  catastrophic 
change  that  may  have  occurred  as  a  result  of  a  natural  or  man-made  disaster.  Rapid 
assessment  of  destructive  events  allows  local  emergency  response  and  local  government 
personnel  to  make  better  actionable  decisions. 

Another  concept  in  LiDAR  and  optical  image  fusion  is  the  ability  to  use  accurate 

stereo  imagery  to  assess  the  quality  of  LiDAR  point  clouds.  In  layman’s  terms,  this  is 

sometimes  known  as  “draping”  the  point  cloud  over  the  stereo  images.  In 

photogrammetric  practice,  a  technique  called  “back-projection”  is  used,  which  takes  the 

X,Y,Z  coordinate  of  each  point  in  the  point  cloud  and  computes  the  corresponding  image 
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coordinates  for  each  point  (Schenk,  Seo,  &  Csatho,  2001).  The  points  can  then  be 
viewed  in  stereo,  comparing  the  point  locations  to  actual  surface  features.  If  the  stereo 
images  have  a  reported  accuracy,  then  this  technique  can  be  used  to  establish  or  verify 
LiDAR  point  cloud  accuracy.  This  technique  is  frequently  used  for  Digital  Elevation 
Model  (DEM)  evaluation. 

The  LiDAR  point  cloud  and  single  optical  image  fusion  process  is  the  most 
intriguing  near  tenn  topic  in  terms  of  what  it  has  to  offer  towards  improved  surface 
analysis.  Standard  practice  for  deriving  precision  3D  coordinates  is  to  use  accurate  stereo 
imagery  and  a  two-ray  image  measurement  intersection  routine.  Performing  a  stereo 
measurement  can  be  very  challenging  for  an  analyst  and  requires  proper  viewing 
equipment.  A  fused  LiDAR-optical  image  data  set  allows  precision  computation  of  3D 
coordinates  without  relying  on  this  stereo  image  measurement.  The  process  of  deriving  a 
3D  coordinate  on  a  single  image  is  called  “single  ray  intersection.”  The  single  ray 
intersection  process  needs  precise  elevation  data  coincident  with  the  single  image  in  order 
to  correctly  intersect  the  image  ray  on  the  ground  or  object.  The  LiDAR  data  registered 
to  the  image  can  provide  these  precise  elevation  data  for  objects  or  ground  features  in  the 
image  allowing  for  precise  3D  coordinate  derivation  through  the  single  ray  intersection 
process  (Rodarmel  et  ah,  2006). 

Being  able  to  compute  precise  3D  coordinates  from  a  single  image  has  advantages 
for  improved  exploitation.  Single  image  exploitation  can  be  done  on  a  handheld  device 
like  a  cell  phone  or  tablet.  The  process  of  exploitation  could  be  as  simple  as  placing  a 
cursor  on  a  feature  in  the  image.  This  research  explores  the  LiDAR  and  optical  image 
fusion  process  starting  with  the  problem  of  image  and  LiDAR  registration. 
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III.  OBJECTIVES 


The  main  objectives  of  this  research  was  to  demonstrate  innovative  techniques  of 
the  LiDAR  point  cloud  and  photogrammetric  image  registration  process  and  to  compare 
LiDAR  point  cloud  and  stereo  photogrammetric  derived  point  clouds,  their 
characteristics,  and  precision.  This  thesis  also  summarizes  the  important  points  of 
creating  a  fused  image/LiDAR  data  set  for  improved  exploitation. 
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IV. 


DATA  PROCESSING  METHODS,  RESULTS,  AND  ANALYSIS 


A.  STUDY  AREA  AND  DATA 

1.  Haiti  LiDAR  and  Optical  Bar  Camera  Imagery 

The  first  study  area  was  chosen  near  Port-au-Prince,  Haiti  because  both  LiDAR 
data  and  Optical  Bar  Camera  (OBC)  imagery  existed  as  a  result  of  support  for  earthquake 
and  humanitarian  relief  efforts.  Both  data  sets  were  collected  in  late  January  of  2010 
following  the  7.0  magnitude  earthquake  that  occurred  near  Port-au-Prince,  Haiti  on 
January  12,  2010.  A  small  subset  of  LiDAR  data  was  selected  from  a  larger  available 
LiDAR  database  that  coincided  with  the  area  of  the  OBC  image  coverage.  The  following 
map  graphics  show  the  exact  location  of  the  LiDAR  subset  (Figure  3). 
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Figure  3.  This  overview  map  (left)  and  zoomed  map  (right)  show  the  area  in  Haiti 
near  Port-au-Prince  where  the  LiDAR  data  were  extracted  from  the 
OpenTopography  Website.  The  red  box  in  the  right  zoom  map  display 
highlights  the  location  of  the  downloaded  LiDAR  data  (3.4  points/m  ). 


The  LiDAR  data  collection  was  performed  by  the  Center  for  Imaging  Science  at 
Rochester  Institute  of  Technology  (RIT)  and  Kucera  International  under  sub-contract  to 
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ImageCat,  Inc.  (OpenTopography,  2008).  The  downloaded  data  consisted  of 
28.5  million  points  with  a  density  of  3.4  points/m  .  The  project  was  funded  by  the  Global 
Facility  for  Disaster  Reduction  and  Recovery  (GFDRR)  and  the  data  are  hosted  at  the 
World  Bank  (www.opentopography.org).  The  following  figure  (Figure  4)  shows  a  3D 
perspective  view  color  coded  by  Above  Ground  Level  (AGL  =  DSM-DEM)  height  of  the 
downloaded  Haiti  LiDAR  data  and  displayed  in  Quick  Terrain  Modeler  (QTM) 
(Applied_Imagery,  2004)  . 


Figure  4.  Oblique  view  of  downloaded  Haiti  LiDAR  data  color  coded  by 
Above  Ground  Level  (AGL)  Height  (in  meters).  This  data  set 
contains  approximately  28.5  million  points. 


The  Optical  Bar  Camera  (OBC)  imagery  is  collected  by  the  United  States  Air 
Force  (UTC  Aerospace  Systems,  2013).  The  OBC  itself  is  a  large  format  calibrated 
panoramic  camera  capable  of  photogrammetric  quality  image  collection.  The  OBC 
imaging  mission  for  Haiti  was  performed  on  January  30,  2010  flying  at  an  altitude  of 
42,000  feet  producing  an  approximate  Ground  Sample  Distance  (GSD)  of  0.15  meters. 
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The  OBC  imagery  is  film  (hardcopy)  based  and  must  be  digitized.  The  large 
format  image  is  traditionally  digitized  using  14  micrometer  pixel  sizes  in  15  degree,  8-bit 
TIF  or  NITF  format  segments.  One  segment  was  chosen  that  covered  the  LiDAR  test 
area  (Figure  3  and  Figure  5).  This  OBC  segment  is  centered  on  the  +15  degree  mark  of 
the  full  (-72  to  +72  degree)  panoramic  image  and  was  further  clipped  to  eliminate  the 
water  area  as  no  control  points  could  be  selected  in  that  area. 
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Figure  5.  Segmented  piece  of  OBC  imagery  in  Haiti  near  the  LiDAR 
coverage  area.  This  image  was  further  clipped  to  eliminate 

the  water. 


2.  NPS  Campus  LiDAR  and  High  Resolution  Aerial  Imagery 

High  resolution  LiDAR  and  optical  data  were  recently  collected  over  a  portion  of 
the  Naval  Postgraduate  School  campus,  Monterey,  California,  and  are  included  as  part  of 

this  study.  The  LiDAR  point  cloud  was  collected  in  October  2012  by  Watershed 
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Sciences,  Inc.  (WSI).  Mounted  in  a  Bell  206  Long  Ranger  Helicopter,  the  LiDAR  data 
were  collected  at  a  very  dense  average  of  30  points/m  using  an  Optech  Orion  C200  laser 
system  (Watershed  Sciences,  2012).  The  following  graphic  shows  the  location  of  the 
complete  LiDAR  collect  over  the  Monterey  Peninsula.  A  subset  of  this  data  over  the 
NPS  campus  was  selected  as  the  focus  of  this  thesis  (Figure  6,  Figure  7). 


Location  of  NPS  Campus  Subset 
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Figure  6.  Image  from  the  Watershed  Sciences  Report  (Watershed  Sciences,  2012) 
showing  the  complete  Monterey  peninsula  WSI  LiDAR  collection 

in  October,  2012 


14 


MSL  Height 
h  36  m 

■  -25  m 


-  25  m 


Sfef 


Location:  36.596  N  121.874  W 


-  14  m 


Scale:  60  m  120  m 


Figure  7.  Selected  LiDAR  subset  of  the  NPS  campus.  3D  Perspective  view 
displayed  in  QT  Modeler  software  and  colored  by  height 
(internally  known  as  tile  774) 

The  optical  imagery  was  collected  in  stereo  using  a  multispectral  UltraCam  Eagle, 
260  megapixel  camera  manufactured  by  Vexel  Corporation,  a  Microsoft  subsidiary.  The 
color  imagery  scans  were  collected  at  450m  flying  height  producing  high  resolution  15cm 
pixels  (Watershed  Sciences,  2012).  The  following  diagram  (Figure  8)  shows  the 
footprints  of  the  single  images,  the  amount  of  overlap  between  successive  images,  and 
the  amount  of  sidelap  between  the  strips.  The  NPS  campus  area  is  near  the  center, 
marked  by  the  red  outline. 
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Figure  8.  UltraCam  Eagle  stereo  image  footprint  coverage  diagram  showing  overlap 

between  images  and  sidelap  between  strips.  The  NPS  campus  area  is  marked  by 

the  red  outline. 


The  imagery  has  very  good  photogrammetric  quality  capable  of  supporting 
precision  DEM  extraction.  The  following  split  screen  view  (Figure  9)  of  the  UltraCam 
Eagle  color  images  shows  a  section  of  the  NPS  campus. 
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Figure  9.  Stereo  images  of  the  NPS  campus  shown  in  split  screen  view  detailing  the 
quality  of  the  stereo  imagery  capable  of  precise,  high  resolution  DEM  or  point 
cloud  extraction.  The  red  lines  highlight  part  of  the  DEM  extraction  area. 


3.  San  Diego  LiDAR  and  WorldView  1  Imagery 

The  third  study  area  was  chosen  near  Pacific  Beach,  California,  which  is  just 
north  of  San  Diego,  California.  This  study  area  was  chosen  because  WorldView  1 
(WV1)  stereo  imagery  was  available  along  with  LiDAR  data  (Open  Topography,  2008). 
A  small  subset  of  LiDAR  data  was  selected  and  downloaded  using  the  opentopography 
web  site  LiDAR  retrieval  tool.  The  LiDAR  was  collected  in  January  2005  by  Merrick, 
Inc.  for  the  City  of  San  Diego  at  1.4  points/m  .  This  small  subset  of  LiDAR  data 
coincided  with  the  area  of  the  WV1  image  coverage.  The  following  map  graphics 
(Figure  10)  show  the  exact  location  of  the  LiDAR  subset. 
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Figure  10.  This  overview  and  zoomed  map  display  show  the  area  in 

Pacific  Beach  near  San  Diego,  California  where  the  LIDAR  data  were 
extracted  from  the  OpenTopography  Website  (OpenTopography,  2008). 

The  red  box  in  the  right  zoom  map  display  highlights  the  location  of  the 
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downloaded  LiDAR  data  (1.4  points/m  ). 
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Figure  1 1 .  Colorized  rendition  of  a  zoomed  in  section  located  in  the  Southeast 
corner  of  the  downloaded  Pacific  Beach  LiDAR  data  set. 

The  data  was  collected  in  2005  at  a  1.4  points/nr  density. 


The  stereo  optical  imagery  used  in  this  third  case  study  is  from  Digital  Globe’s 
WV1  satellite.  WV1  is  strictly  a  panchromatic  (B/W)  image  collector  with  large 
footprints  and  0.6m  pixels.  The  images  were  collected  as  a  stereo  triplet — 3  images 
consecutively.  Each  stereo  pair  image  combination  (1-2,  1-3,  2-3)  from  the  WV1  triplet 
was  used  to  extract  a  DEM  using  photogrammetric  methods,  and  from  these  three 
independent  DEMs,  the  one  with  the  least  amount  of  automatic  correlation  errors  was 
used.  The  following  graphic  (Figure  12)  shows  the  coverage  of  the  WV1  stereo  triplet  as 
well  as  the  outline  of  the  LiDAR  coverage. 
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Figure  12.  This  image  shows  the  large  red  outline  of  San  Diego WV1  stereo 
triplet  coverage  and  a  smaller  interior  red  box  that  outlines  the 
downloaded  LiDAR  coverage  for  the  Pacific  Beach  area. 


The  following  images  (Figure  13)  show  a  split  screen  view  of  the  WV1  stereo 
imagery  detailing  the  quality  and  perspective  view  of  the  individual  images. 
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Figure  13.  WV-1  stereo  image  area  of  Pacific  Beach  shown  in  split  screen  view 

detailing  the  quality  of  the  stereo  imagery  capable  of  DEM  or  point  cloud 
extraction.  (GSD  =  0.6  meters) 

B.  REGISTRATION  OF  OPTICAL  IMAGERY  TO  LIDAR 

This  research  focused  on  using  the  “Intensity  Image”  approach  for  controlling 
optical  imagery  to  LiDAR  (Mishra  &  Zhang,  2012)  as  opposed  to  the  feature  or  edge 
matching  approach  that  has  been  thoroughly  studied  by  several  researchers  (Du,  Xu,  & 
Cao,  2009;  Mishra  &  Zhang,  2012).  In  the  majority  of  cases,  the  identifiable  feature 
content  of  an  intensity  image  will  be  sufficient  enough  for  common  points  to  be  found 
between  the  optical  image  and  the  intensity  image.  It  is  also  important  to  select  these 
common  points  in  relatively  flat  regions  because  the  height  value  for  the  point  selected  on 
the  intensity  image  will  have  to  be  interpolated  from  the  LiDAR  created  TIN  or  DEM.  If 
the  point  selection  is  done  properly,  very  accurate  points  can  be  interpolated  from  the 
LiDAR  data  even  though  the  selected  point  may  not  represent  a  real  collected  LiDAR 
point.  Once  collected,  these  precision  interpolated  points  can  be  used  to  control  the 
optical  imagery  in  a  triangulation  process. 


21 


The  following  process  flow  summarizes  a  standard  method  for  controlling  optical 
imagery  to  LiDAR  data  (Table  1).  This  generic  process  was  derived  from  similar  image- 
to-image  registration  processes  using  the  “Socet  Set”  software  (BAE_Systems,  1990). 


Generic  LiDAR  Control  Process  Flow 

1.  Import  LiDAR  data  (.las  file) 

A.  Creating  Intensity  Image 

B.  Creating  DEM  (no  thinning) 

2.  Import  Imagery  to  be  controlled 

3.  Build  Control  Point  File  using  Control  Point  Editor 

A.  Locate  Control  Points  on  Intensity  Image 

B.  Interpolate  Height  using  LiDAR  DEM  (Terrain  Tracking) 

C.  Capture  a  description  of  control  point  location  (sketch) 

4.  Create  Triangulation  File 

A.  Include  Image  and  Control  Point  File 

5.  Locate  and  Measure  each  Control  Point 
A.  Use  sketch  for  correct  identification 

6.  Use  Triangulation  Solve  to  adjust  Imagery 


Table  1.  Generic  LiDAR  Control  Process  Flow  Defined  Using  Socet  Set 


As  part  of  this  research  an  innovative,  improved  process  was  developed  to  control 
imagery  to  LiDAR  (Table  2). 
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Improved  LiDAR  Control  Process  Flow 

1.  Import  LiDAR  data  (.las file) 

A.  Creating  Intensity  Image 

B.  Creating  DEM  (no  thinning) 

2.  Import  Imagery  to  be  controlled 

3.  Build  Triangulation  File 

A.  Include  Imagery  and  Intensity  Image 

4.  Measure  Common  Points 

A.  On  LiDAR  Intensity  Image  and  Optical  Imagery 

B.  Point  Pattern  Sufficient  for  Image  Type 

5.  Create  Control  Points  from  LiDAR  Data  Using  Triangulation  Solve 

A.  Solve  for  ground  point  locations  on  intensity  image 

B.  Update  Ground  Control  File 

C.  Interpolate  Height  using  LiDAR  DEM 

D.  Update  ground  control  file  with  new  height 

6.  Use  Triangulation  Solve  to  adjust  Imagery 

Table  2.  Improved  LiDAR  Control  Process  Flow  Derived  Using  Socet  Set 


The  improved  LiDAR  control  process  contains  two  enhancements.  First, 
including  all  imagery  in  the  Triangulation  allows  common  points  to  be  measured  on  all 
imagery  in  the  same  viewing  window  and  allows  the  review  of  those  measured  points  at 
any  time  before  or  after  the  triangulation  solution.  The  capability  to  review  points  is  very 
useful  in  the  case  of  control  point  miss-identification.  Second,  by  activating/deactivating 
images,  changing  weights  on  image  parameters,  and  changing  weights  on  ground  control 
points,  the  triangulation  solve  can  be  used  to  derive  ground  control  points  in  one  step  and 
then  used  to  adjust  the  optical  imagery  in  a  second  step.  Using  the  triangulation  solve  to 
generate  ground  control  also  gives  a  triangulation  report  that  details  residuals  and 
parameter  adjustments  that  can  be  used  for  quality  assurance. 


23 


1.  Haiti  LiDAR  and  Optical  Bar  Camera  Imagery 

Once  the  basics  of  registering  optical  imagery  to  LiDAR  were  outlined  as  above, 
the  first  challenging  case  study  was  to  investigate  the  registration  of  Optical  Bar  Camera 
(OBC)  imagery  to  LiDAR  for  the  Haiti  site.  The  registration  of  OBC  imagery  using 
control  points  is  challenging.  The  challenging  part  is  that  the  imagery  were  collected 
with  an  airborne  panoramic  camera,  which  introduces  the  complicated  dynamics  of  the 
panoramic  camera  and,  in  the  case  of  OBC,  disconnected  sensor  positioning  and 
orientation  information.  Also,  OBC  imagery  starts  out  as  a  hardcopy  film  that  must  be 
digitized.  Importing  this  scanned  imagery  into  photogrammetric  exploitation  software 
using  the  correct  panoramic  math  model  and  proper  metadata  can  be  tedious. 

The  required  sensor  position  and  orientation  information  is  separate  from  the 
imagery  and  must  be  correctly  identified  and  captured.  Once  this  important  data  is 
retrieved,  the  values  must  be  hand  entered  in  required  fields  upon  image  import.  The 
example  below  is  from  the  Socet  Set  software  (Figure  14). 


Figure  14.  Socet  Set  panoramic  camera  position  and  orientation  input  page. 

All  data  must  be  input  by  hand. 
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Because  the  scanned  imagery  is  a  segment  of  a  full  panoramic  image  that  may 
occur  at  any  degree  sweep,  the  input  roll  angle  of  the  sensor  has  to  be  changed  to  give  the 
impression  that  the  camera  is  pointing  in  that  direction  (Figure  15). 


Figure  15.  Diagram  of  a  15  degree  scanned  segment  of  OBC  film  centered 

at  the  negative  1 5  degree  scan  mark. 

If  the  metadata  are  correctly  input,  the  imagery  will  import  properly  and  align  to 
within  100  meters  of  absolute  ground  truth.  In  any  case,  this  raw  OBC  image  positioning 
otherwise  known  as  “apriori  positioning”  requires  controlling  or  registration  for  LiDAR 
fusion  and  exploitation.  At  this  juncture,  the  LiDAR  data  are  also  imported,  creating  an 
intensity  image  and  a  regularly  spaced  gridded  DEM  interpolated  from  the  LiDAR  point 
cloud.  All  of  the  ingredients  are  now  in  place  to  continue  the  optical  image  to  LiDAR 
registration  process  as  outlined  in  Table  2. 

The  next  step  in  the  process  is  to  measure  common  points  between  the  OBC 
imagery  and  the  LiDAR  intensity  image  using  an  interactive  point  measurement  function. 
Selecting  and  measuring  quality  common  points  are  a  skill  that  requires  practice.  The 
goal  in  optical  image  to  LiDAR  intensity  image  point  transfer  is  to  find  small  features 
that  are  clearly  identifiable  in  both  images.  Also,  because  we  need  to  interpolate  a  height 
value  from  the  LiDAR  DEM,  the  selected  point  should  be  in  a  relatively  flat  area.  Figure 
16  shows  an  example  of  a  common  selected  point. 
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Figure  16.  This  side-by-side  image  display  of  a  subset  of  the  Haiti  data  shows  an 
example  of  a  common  point  selected  (in  center)  between  the  optical  image 
(Left:  OBC)  and  the  LiDAR  intensity  image  (Right:  Haiti  LiDAR  Intensity 
Image).  Points  should  be  selected  that  are  clearly  identifiable  and  do  not  have 
variation  in  surrounding  height  values. 


Once  all  the  common  registration  points  were  measured  in  a  pattern  that  covers 
the  image  and  dense  enough  for  sufficient  redundancy  to  solve  the  adjustment  parameters 
(see  Figure  17  for  example  of  a  sufficient  pattern),  the  control  point  X  and  Y  (Latitude 
and  Longitude  if  working  in  geographical  space)  were  derived  from  the  LiDAR  intensity 
image.  This  was  accomplished  by  performing  a  single  image  triangulation  adjustment 
with  the  image  parameters  held  fixed  and  the  Z  ground  space  coordinate  held  fixed.  The 
Z  height  value  is  an  arbitrary  value  at  this  point  and  it  does  not  matter  what  the  value  is 
because  the  LiDAR  intensity  image  is  an  orthoimage. 

After  the  initial  X  and  Y  values  for  the  control  points  were  determined  from  this 
first  triangulation  solution,  the  Z  height  value  was  updated  using  the  control  point  editor. 
The  correct  Z  height  value  was  detennined  by  turning  on  a  terrain  tracking  tool,  selecting 
the  LiDAR  DEM  as  the  base  terrain  layer,  systematically  visiting  each  control  point  X 
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and  Y  location,  and  recording  the  Z  height  value.  This  step  completed  the  creation  of  full 
X,Y,Z  control  points  to  be  used  to  register  the  OBC  image. 

The  final  triangulation  solution  that  adjusted  the  OBC  imagery  using  the  rigorous 
panoramic  math  model  parameters  produced  alignment  to  the  LiDAR  point  cloud  data 
with  marginally  acceptable  accuracy  (Figure  17).  The  Root  Mean  Square  (RMS)  of  the 
ground  residuals  was  1.9  meters  with  the  longest  residual  at  3.0  meters.  This  corresponds 
to  several  pixels  of  misalignment  for  the  image  and  LiDAR  data. 

Previous  to  this  research,  investigation  had  been  done  to  develop  more  robust  and 
user  friendly  support  data  for  OBC  imagery  that  would  have  more  utility  and  be  able  to 
be  used  across  many  software  packages,  not  just  Socet  Set.  Success  had  been  achieved  in 
computing  RPC  support  data  for  each  segment,  which  made  exploitation  much  easier. 
An  added  benefit  surfaced  as  a  result  of  the  comparisons  between  the  rigorous  model 
support  data  and  the  RPC  support  data.  The  added  benefit  of  the  RPC  support  data  was 
that  when  the  RPC  adjustable  parameters  were  used  to  adjust  the  imagery,  a  better  fit  to 
control  was  achieved. 

Because  of  this  previous,  unpublished  knowledge,  RPC  support  data  was 
generated  for  the  Haiti  segment  using  the  Socet  Set  “fast  sensor  model”  and  “image 
chipping  tool.”  The  same  control  point  selection  approach  was  then  used  to  register  OBC 
to  the  LiDAR  with  very  good  results  (Figure  17).  The  RMS  of  the  ground  residuals 
improved  to  0.5  meters  with  the  longest  ground  residual  at  1.1  meters. 
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Figure  17.  These  base  images  for  Haiti  show  a  scanned  segment  of  an  OBC  image 

and  the  control  point  locations  (red  circle  with  red  diamond)  with  red  vectors. 
The  Left  image  shows  ground  space  residuals  as  a  result  of  the  rigorous 
panoramic  model  adjustment.  (Left:  RMS  1.9  meters) 

The  Right  image  shows  the  improved  ground  space  residuals  as  a  result  of  using 
the  RPC  adjustment  (Right:  RMS  0.5  meters).  Scale:  Longest  vector  is  1 . 1 

meters  on  the  right  image. 


Following  this  RPC  parameter  adjustment  aligning  the  OBC  imagery  to  the 
LiDAR  point  cloud,  an  orthorectification  of  the  OBC  imagery  was  performed  using  the 
adjusted  RPC  parameters  and  the  LiDAR  DEM.  QT  Modeler  software  was  then  used  to 
merge  the  LiDAR  point  cloud  and  the  OBC  orthophoto  in  a  fusion  process.  The  process 
in  QT  Modeler  simply  imports  the  LiDAR  data  and  then  adds  the  orthophoto  as  texture. 
Internal  to  QT  Modeler,  the  LiDAR  data  is  stored  as  individual  XYZ  points  with  each 
point  receiving  the  added  attribute  of  a  Red-Green-Blue  (RGB)  value  derived  from  the 
OBC  orthophoto  at  that  pixel  location.  The  following  figure  (Figure  18)  shows  the 
results  of  the  LiDAR  point  cloud  and  OBC  image  fusion  using  QT  Modeler. 
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Figure  18.  OBC  image  and  LiDAR  point  cloud  fusion  showing  the  complete 

registered  image.  LiDAR  points  are  color  coded  by  Above  Ground  Level  (AGL) 

height  where  AGL  =  DSM  -  DEM. 


The  following  figure  (Figure  19)  shows  a  detailed  perspective  view  using  QT 
Modeler  to  display  the  LiDAR  point  cloud  with  the  OBC  image  used  as  texture  for  the 
points. 
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Figure  19.  Haiti  LiDAR  data  with  the  registered  OBC  used  as 

texture  for  the  points 


Figure  19  demonstrates  that  the  building  tops  appear  to  have  good  alignment  and 
that  the  road  and  tower  on  the  hill  also  have  good  alignment  with  the  OBC  image  texture. 
Individual  points  from  the  point  cloud  can  also  be  seen  on  the  figure. 

2.  NPS  Campus  LiDAR  and  High  Resolution  Aerial  Imagery 

The  NPS  Campus  LiDAR  and  High  Resolution  Aerial  Imagery  were  collected  in 
conjunction  with  precision  GPS  and  INS/IMU  data  to  meet  accuracy  requirements. 
Because  of  this  accuracy  requirement,  no  registration  was  performed  to  adjust  the 
imagery  to  the  LiDAR  point  cloud  data. 

3.  San  Diego  LiDAR  and  WorldView  1  Imagery 

The  WV1  stereo  image  registration  to  LiDAR  used  the  same  LiDAR  “Intensity 
Image”  and  LiDAR  DEM  interpolation  technique  to  create  control  points  that  were  used 
in  a  WV1  only  triangulation  process.  Refer  to  Table  1  above  for  the  high  level  process 
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flow.  The  following  diagram  shows  the  location  of  the  control  points  in  the  area  of  the 
LiDAR  data  set. 


Scale: 


4  km 


Location:  32.810  N  117.257  W 


Figure  20.  WV 1  background  image  (image  214)  showing  the  location  of  the 

LiDAR  boundary  in  red  together  with  the  location  of  the  control  points 
(12  points  identified  as  red  triangles  within  red  circles) 


Control  points  on  the  LiDAR  intensity  image  were  selected  in  relatively  flat, 
elevation  homogeneous  regions.  Many  of  the  points  were  small,  flat  features  in  the 
center  areas  on  roof  tops  as  this  test  area  was  primarily  urban.  The  following  example 
shows  the  selection  of  a  control  point  on  a  street  marking.  Generally,  painted  street 
markings  do  not  make  good  image  to  image  registration  control  points  because  they  can 
change  or  be  moved  slightly  over  a  short  time  span.  This  one  appeared  to  be  relatively 
unchanged. 
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Figure  2 1 .  This  side-by-side  image  display  shows  an  example  of  a  common  point 
selected  (center  red  crosshair)  between  the  optical  image  (Left:  WV1) 
and  the  LiDAR  intensity  image  (Right:  Pacific  Beach  LiDAR  Intensity  Image). 

Points  should  be  selected  that  are  clearly  identifiable  and  have  very  little  variation 

in  surrounding  height  values 

Once  all  the  control  points  are  selected,  the  horizontal  ground  coordinates  for  the 
points  on  the  intensity  image  must  be  computed.  For  the  purposes  of  this  research,  this 
was  accomplished  using  the  Socet  Set  solve  routine  and  adjusting  only  the  X  and  Y  value 
of  the  point.  It  could  also  be  computed  in  other  software  using  similar  approaches.  The 
image  position  parameters  are  “locked”  and  the  Z  coordinate  (meaningless  in  this  initial 
adjustment)  is  also  “locked.”  Running  the  solve  with  these  settings  and  only  the  intensity 
image  active  will  give  precise  horizontal  coordinates  for  the  points  that  now  become 
control  points. 

The  vertical  control  value  is  arrived  at  through  interpolation  of  the  LiDAR  DEM, 
as  the  point  that  has  been  created  does  not  coincide  with  a  real  point  in  the  LiDAR  point 
cloud.  In  Socet  Set,  this  can  be  accomplished  by  turning  on  terrain  tracking  with  the 
LiDAR  DEM  selected  and  systematically  visiting  each  point.  Similar  approaches  can  be 
used  in  other  software.  This  will  allow  interpolation  of  a  Z  value  for  the  visited  control 
X,Y  location.  These  interpolated  Z  values  now  become  the  precise  vertical  control 
location. 
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The  final  controlling  adjustment  on  the  WV1  stereo  images  is  performed  using 
these  control  points  and  a  normal,  WV1  only  adjustment.  Results  of  this  adjustment  were 
very  good  as  shown  in  Figure  22.  Ground  control  points  fit  to  an  RMS  of  0.6  meters. 


Figure  22.  WV 1  background  image  showing  ground  space  residual  analysis  results 

of  the  triangulation  adjustment  that  controlled  the 
WV-1  stereo  triplet  to  the  LiDAR  data.  The  RMS  was  0.6  meters 


C.  STEREO  IMAGE  POINT  CLOUD  EXTRACTION 

The  next  critical  step  in  this  process  was  to  extract  a  digital  surface  model  (DSM) 
from  the  stereo  images.  Socet  Set  has  an  automated  dense  terrain  extraction  tool  called 
the  Next  Generation  Automated  Terrain  Extraction  (NGATE)  tool.  NGATE  has  many 
features  that  allow  dense  DSMs  to  be  automatically  measured  that  resemble  a  LiDAR 
point  cloud.  One  feature  is  the  interpolated  estimate  or  starting  point  of  adjacent  points. 
The  software  will  use  surrounding  point  correlation  information  to  better  predict  the 
height  to  try  an  auto-correlated  measurement.  A  second  feature  is  the  ability  of  the 
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software  to  take  advantage  of  multiple  (more  than  2)  perspective  views  of  a  single  point. 
This  can  work  well  around  buildings  and  occluded  objects.  It  should  be  possible  to  use 
similar  approaches  to  obtain  similar  results  using  other  DSM-generation  software. 

1.  Haiti  Site  (OBC) 

No  stereo  data  were  available  for  the  Haiti  site,  thus  a  stereo  point  cloud  was  not 
attempted. 

2.  NPS  Campus  Stereo  Imagery 

The  NPS  campus  stereo  imagery  is  very  good  quality  photogrammetric  imagery 
as  reported  by  the  WSI  report  (Watershed  Sciences,  2012)  and  as  shown  in  Figure  8  and 
Figure  9  above.  The  stereo  pair  that  covered  the  “774”  subset  of  the  NPS  campus  area 
was  image  id  105  and  106.  These  images  have  an  average  of  0.12  meters  ground  sample 
distance  (GSD).  NGATE  was  set  up  to  produce  a  grid  spacing  equal  to  that  of  the 
LiDAR  collection  at  0.15  meters.  The  following  figure  (Figure  23)  shows  an  example  of 
the  DEM  grid  spacing.  Even  though  this  is  a  very  small  area,  the  DEM  grid  contains 
4,020,025  posts.  Every  fifth  post  has  been  displayed  in  order  to  see  the  grid  better. 
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Figure  23.  Portion  of  the  DEM  grid  (every  5  th  post  displayed)  near  Herrmann  Hall 

on  the  NPS  campus  showing  the  density  of  the  grid 
(0. 15  meter  point  spacing) 

Fortunately,  the  NGATE  software  in  Socet  Set  will  attempt  autocorrelation  on  all 
points  as  it  would  be  virtually  impossible  to  edit  each  point  manually.  The  results  of  the 
collected  NGATE  autocorrelation  DEM  is  shown  in  Figure  24.  It  should  be  possible  to 
generate  similar  results  using  other  autocorrelation  software. 


35 


Figure  24.  High  density  DEM  (0.15  meter  point  spacing)  over  NPS  campus 

auto  collected  from  stereo  pair  105/106  using  Socet  Set  NGATE  software 
(4,020,025  total  points).  These  data  were  be  used  for  LiDAR/Optical  fusion. 
MSL  =  Mean  Sea  Level  heights  (MSL  =  Ellipsoid  Height  -  Geoid  Separation) 


The  next  step  in  the  process  was  to  take  the  high  density  DEM  and  convert  it  to  a 
point  cloud.  This  was  accomplished  by  exporting  the  DEM  from  Socet  Set  as  an  ASCII 
grid  file  and  importing  the  ASCII  grid  file  into  QT  Modeler  as  a  point  cloud.  The 
following  figure  (Figure  25)  shows  the  high  density  collected  DEM  converted  to  a  point 
cloud.  Other  software  should  be  able  to  achieve  similar  results. 
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Figure  25.  High  density  DEM  (0.15  m.  post  spacing)  imported  into  QT  Modeler 

as  a  point  cloud  to  be  used  for  LiDAR  fusion. 


At  first,  it  appeared  as  though  there  were  too  many  errors  around  the  trees  and 
near  the  building  edges  of  the  NGATE  autocorrelation.  The  auto-correlated  points  in  the 
tree  and  building  edge  areas  seemed  to  be  interpolated  and  of  poor  quality.  Including 
3  images  (104/105/106)  in  the  stereo  extraction  was  attempted  to  aid  in  precise 
autocorrelation.  The  results  did  not,  however,  improve  the  stereo  point  cloud  as  well  as 
expected  (Figure  26).  Many  autocorrelation  errors  still  existed.  The  autocorrelation 
errors  were  also  affected  by  sun  shadows  even  though  these  images  were  all  shot  within 
tens  of  seconds  of  each  other,  and  the  shadows  did  not  change. 
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Location:  36.596  N  121.874  W 


Figure  26.  Side  by  side  view  showing  the  difference  between  a  3-photo  stereo 
extraction  (Left  Image)  and  a  2-photo  stereo  extraction  (Right  Image). 
Autocorrelation  errors  exist  (10  meters)  around  trees  and  near  building 
edges  in  both  images  with  no  discernible  advantage  for  either  method. 

The  two  photo  stereo  point  cloud  was  considered  acceptable  for  use  in  the  LiDAR 
fusion  process  because  all  other  stereo  image  autocorrelation  results  were  similar.  A 
realization  did  surface  from  this  experiment.  It  was  concluded  that  any  level  of  stereo 
DEM  extraction  would  contain  errors.  The  consequence  is  that  the  data  have  to  be 
manually  edited  by  measuring  each  point  individually  or  using  tools  to  edit  points  as  a 
group.  This  can  be  extremely  time  consuming  and  expensive.  The  alternative  is  to 
accept  the  errors,  and  use  the  data  in  the  unedited  state — pure  autocorrelation  data — 
which  is  the  path  that  I  chose  for  this  research. 

3.  San  Diego  WV-1  Stereo  Imagery 

The  WV-1  stereo  imagery  over  the  San  Diego  area  (Figure  12  and  Figure  13 
above)  was  collected  as  what  is  called  a  stereo  triplet.  This  stereo  triplet  is  three  images 
collected  consecutively  in  the  same  orbital  pass  over  an  area.  WV-1  imagery  when 
collected  has  absolute  position  capability  to  6  meters  90%  of  the  time  and  each  image  is 
independent.  The  fact  that  the  images  are  independent  implies  that  the  stereo  pairs  will 
have  parallax  that  must  be  removed.  This  removal  of  parallax  and  the  absolute 
positioning  improvement  was  accomplished  in  one  step  when  all  three  WV-1  images 
were  registered  to  the  LiDAR  data  (Figure  22  above).  The  following  figure  (Figure  27) 
shows  the  perspective  views  of  the  stereo  triplet. 
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Figure  27.  Side  by  side  display  of  the  WV-1  triplet  zoomed  in  on  a  prominent 

building  feature.  The  images  were  collected  during  the  same  orbital  pass 
on  08  Feb  2009.  From  left  to  right  the  image  id’s  are  139,  214,  and  228 

Images  214  and  228  were  chosen  for  stereo  image  DEM  extraction  because  the 
perspective  views  were  similar  resulting  in  less  occluded  features.  Figure  27 
demonstrates  that  image  139  has  a  perspective  view  that  is  completely  different  than 
image  214  and  228.  Using  image  139  with  any  other  image  would  prove  to  be  very 
difficult  for  the  autocorrelation  routine. 

The  grid  spacing  for  the  WV-1  stereo  image  extracted  DEM  was  set  similar  to  the 
LiDAR  data  point  spacing  at  1.4  points  per  meter.  This  translated  into  approximately  0.8 
meter  equivalent  grid  spacing  for  the  DEM.  This  is  about  the  resolution  of  the  WV-1 
imagery  at  0.7  meter  GSD.  The  following  figure  (Figure  28)  shows  the  DEM  grid 
displayed  over  the  WV-1  image  (every  5th  point  has  been  displayed  for  clarity). 


Figure  28.  Portion  of  the  WV-1  stereo  derived  DEM  grid  (every  5th  post  displayed) 
showing  the  density  of  the  grid  (0.8  meter  point  spacing) 

The  complete  collection  of  the  Socet  Set  NGATE  automatic  DEM  point 
extraction  is  shown  in  the  following  figure  (Figure  29). 
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Figure  29.  WV-1  stereo  pair  collected  DEM  (0.8  meter  point  spacing)  auto 

collected  from  stereo  pair  214/228  using  Socet  Set  NGATE  software 
(2,141,400  total  points). 

The  next  step  in  the  process  was  to  take  this  stereo  derived  WV-1  DEM  and 
generate  a  point  cloud.  This  was  accomplished  by  exporting  the  point  cloud  as  an 
American  Standard  Code  for  Information  Interchange  (ASCII)  DEM  and  importing  this 
ASCII  DEM  into  QT  Modeler  as  a  point  cloud.  The  following  figure  (Figure  30)  shows  a 
display  of  that  point  cloud. 
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Figure  30.  WV-1  stereo  derived  DEM  converted  to  a  point  cloud  to  be  used 

for  LiDAR  fusion. 


Similar  to  the  high  resolution  image  DEM  extraction  from  aerial  photographs,  the 
WV-1  point  cloud  has  many  errors  near  building  edges  and  near  trees.  No  manual  editing 
was  performed  and  the  data  were  used  as  a  pure  autocorrelation  data  set.  At  this  juncture, 
the  DEM  point  cloud  data  was  ready  for  LiDAR  fusion. 

D.  CHANGE  DETECTION  USING  LIDAR  AND  STEREO  POINT  CLOUDS 

Fusion  of  LiDAR  point  cloud  data  and  stereo  point  cloud  data  has  many 
applications  as  outlined  in  the  Introduction  and  Background  chapters  of  this  thesis.  This 
research  focused  on  two  of  those  applications  in  the  form  of  point  cloud  change  detection 
and  a  fused  Image-LiDAR  product  capable  of  measuring  precision  3D  coordinates.  For 
change  detection  to  occur,  the  LiDAR  point  cloud  and  stereo  image  point  cloud  data 
sources  must  be  coincident.  The  previous  sections  in  this  thesis  highlight  methods  to 
achieve  this  coincidental  alignment  by  using  the  LiDAR  intensity  image  and  LiDAR 
DEM  as  a  control  source  for  the  stereo  imagery.  The  registration  of  the  stereo  imagery  to 
the  LiDAR  data  results  in  a  stereo  derived  DEM  that  is  coincident  with  the  LiDAR  point 
cloud. 


42 


Change  detection  can  be  an  important  tool  to  rapidly  assess  damage  due  to  a 
natural  or  man-made  event.  Natural  events  like  hurricanes,  earthquakes,  tsunamis,  or 
fires  can  be  assessed  with  before  event  and  after  event  feature  level  data  like  a  LiDAR 
point  cloud  or  stereo  derived  point  cloud.  The  following  examples  of  the  NPS  campus 
data  and  the  Pacific  Beach  data  demonstrate  how  change  detection  can  be  used  for 
analysis  in  spite  of  potential  noise  in  the  data  sets. 

1.  NPS  Campus  LiDAR  and  Stereo  point  cloud  change  detection 

Before  beginning  the  change  analysis,  it  was  important  to  inspect  and  visualize 
the  data.  The  following  figure  (Figure  31)  show  a  side-by-side  view  of  the  LiDAR  point 
cloud  and  the  Stereo  point  cloud  for  comparison  of  quality. 
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Figure  3 1 .  Split  screen  close  up  view  of  the  LiDAR  data  on  the  left  and  the 

stereo  image  collected  point  cloud  on  the  right  showing  many  unique  differences. 
The  LiDAR  data  is  very  clean  and  sharp  (Left)  and  the  Stereo  point  cloud  is 
interpolated  and  rough  in  areas  (Right). 


It  proved  to  be  very  difficult  to  get  a  good,  clean  point  cloud  from  the  optical 
imagery  even  though  the  imagery  was  high  quality.  Notice  how  well  the  LiDAR  point 
cloud  data  modeled  the  trees  as  can  be  seen  by  the  realistic  display  of  the  tree  in  the 
center  of  the  left  image  of  Figure  31.  Also,  notice  how  clean  the  building  comer  lines  are 
for  the  LiDAR  (Figure  31,  center  red  building  on  left  image)  and  how  noisy  and 
sometimes  interpolated  the  building  lines  are  in  the  stereo  cloud  (Figure  31,  center  red 
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building  on  right  image).  Stereo  image  autocorrelation  in  tree  canopy  is  very 
challenging.  As  was  mentioned  before,  the  stereo  point  cloud  data  was  not  manually 
edited  but  used  as  a  pure  autocorrelation  data  set. 

The  next  step  was  to  come  up  with  a  method  to  compare  these  two  data  sets  for 
change  detection  in  spite  of  the  differences  already  noticed  from  the  autocorrelation 
errors.  A  straight  subtraction  method  was  used  (data  set  1  minus  data  set  2)  to  create  a 
new  point  cloud  model  for  display  and  analysis.  Figure  32  shows  the  results  of  the  new 
point  cloud  created  by  subtracting  the  LiDAR  point  cloud  from  the  stereo  point  cloud. 
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Figure  32.  Difference  between  the  WSI  LiDAR  and  Stereo  Imagery  point 
cloud  Mean:  -0.5m;  Std  Dev:  6.3m;  Max:  44.0m;  Min:  -30.7m 


Results  appear  very  noisy  but  predictable.  The  trees  and  building  edges  have  the 
most  noticeable  differences  as  indicated  by  the  red  and  blue  coloration.  Even  though  the 
differences  look  catastrophic,  there  are  some  good  things  in  here.  For  instance,  building 
tops,  like  the  ground  surface,  have  not  changed. 

The  results  of  the  autocorrelation  and  the  errors  that  could  potentially  exist  on 

either  data  set  required  approaching  the  problem  from  a  different  perspective.  Instead  of 

focusing  on  the  differences,  focusing  on  the  things  that  stayed  the  same  seemed  like  a 

better  approach.  A  high  percentage  of  the  autocorrelation  data  was  very  good  with 
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autocorrelation  performing  well  on  the  ground  features  and  building  tops.  The  new 
approach  strategy  was  to  keep  the  data  that  were  similar  and  eliminate  the  noisy  outlier 
data.  While  there  are  a  number  of  ways  to  implement  this,  the  above  approach  was 
accomplished  for  this  research  by  using  software  called  Cloud  Compare  (CloudCompare, 
2004)  and  clipping  the  difference  cloud  that  was  created  in  QT  Modeler.  The  following 
figure  (Figure  33)  is  a  visualization  of  the  original  difference  cloud  in  the  Cloud  Compare 
software. 


Figure  33.  Vertical  display  of  the  point  cloud  created  from  the  difference  between 
the  NPS  WSI  LiDAR  and  the  Stereo  Image  autocorrelation  point  cloud 
using  Cloud  Compare  (CC)  software. 

Notice  how  the  majority  of  the  cloud  differences  shown  in  figure  33  are  near  zero 
(green).  The  analysis  focused  on  how  much  of  the  scene  stayed  the  same  and  used  a 
difference  histogram  to  capture  the  statistics.  The  scale  bar  shows  the  height  difference 
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values  (in  meters)  compared  to  the  colors.  A  small,  “on  edge”  vertical  histogram  to  the 
right  of  the  scale  bar  highlights  the  fact  that  a  majority  of  the  differences  are  biased  by 
about  one  meter. 

The  following  figure  (Figure  34)  represents  only  those  objects  that  remain  the 
same.  The  technique  is  to  remove  the  change  outliers,  keeping  only  things  that  stayed  in 
the  same  location.  The  change  outliers  are  extreme  differences  caused  by  stereo  model 
auto-correlation  inaccuracies,  sensor  perspective  differences,  and  normal  time  variant 
change  (cars,  building  construction,  tree  removal,  etc.). 
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Figure  34.  Display  of  the  first  cut  edit  of  the  difference  data  clipped  at  plus  and 
minus  2  meters  to  remove  large  outliers.  The  “holes”  in  the  data  are 
where  outlier  errors  have  been  removed. 


The  color  image  and  color  scale  bar  show  how  much  things  are  the  same  between 


the  LiDAR  point  cloud  and  the  Stereo  Image  derived  point  cloud.  The  difference  data 
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have  been  clipped  at  plus  and  minus  2  meters  from  the  mean  to  remove  the  outliers.  The 
“holes”  are  where  the  outliers  existed.  Amazingly,  34%  of  the  points  have  been  clipped 
but  the  contents  of  the  scene  are  still  discernible  (Figure  34). 

By  further  clipping  the  outlier  free  data  using  the  statistical  95%  confidence 
interval  equation  (mean  plus  or  minus  two  times  the  standard  deviation),  subtle 
differences  are  detected  -  almost  at  the  sensor  characteristic  level. 
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Figure  35.  Display  of  the  difference  data  after  further  clipping  at  the 
95%  confidence  level.  The  differences  are  on  the  order 
of  50  cm  (.5  meters)  from  the  mean  and  show  peculiar  sensor  collection 
level  artifacts  (red  and  blue  areas). 
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In  Figure  35,  notice  the  50  cm  difference  (red)  in  the  driveway  on  the  upper  right 
of  the  image  (Location  1  identified  with  the  red  arrow).  Next,  there  is  a  -50  cm 
difference  (blue)  in  the  walkway  on  extreme  upper  right  of  the  image  (Location  2 
identified  with  the  red  arrow).  Notice  that  the  southeast  facing  roof  sections  of  buildings 
show  a  50  cm  difference  (Location  3  identified  with  the  red  arrow).  These  are  all 
examples  of  extremely  small  differences  that  are  sensor  collection  dependent  or  stereo 
extraction  dependent. 

The  important  point  is  that  the  buildings  and  many  of  the  trees  remained  the  same 
even  though  quite  a  bit  of  noise  due  to  errors  in  stereo  correlation  existed.  Focusing  on 
things  that  remained  the  same  by  clipping  the  gross  error  outliers  has  allowed  analysis  to 
take  place  at  whatever  level  the  errors  exist  at.  The  data  showed  that  no  extreme 
differences  had  occurred.  The  interesting  thing  about  this  analysis  was  that  the  final 
clipped  difference  data  set  (90%  confidence  interval)  was  able  to  find  differences  at  the 
sensor  collection  level. 

2.  San  Diego  LiDAR  and  Stereo  WV1  point  cloud  change  detection 

Visualization  and  inspection  of  the  data  was  performed  in  a  similar  fashion  to  that 
performed  with  the  NPS  data.  The  following  figure  (Figure  37)  shows  a  side-by-side 
comparison  of  the  LiDAR  point  cloud  and  WV-1  stereo  extracted  point  cloud. 


Figure  36.  Side-by-side  zoomed  in  view  of  the  Pacific  Beach,  CA, 

LiDAR  point  cloud  (left)  and  WV-1  stereo  point  cloud  (right). 
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The  LiDAR  point  cloud  is  clean  and  sharp  while  the  stereo  point  cloud  shows 
areas  of  interpolation  and  missing  features.  As  rough  as  the  WV-1  stereo  extracted  point 
cloud  was,  it  was  used  as  a  pure  autocorrelation  data  set  in  the  following  analysis. 

The  following  figure  (Figure  38)  is  the  “first  cut”  removal  of  the  outliers  using  a 
plus  and  minus  10  foot  threshold  around  the  approximate  minus  4  foot  bias.  Fifteen 
percent  (15%)  of  the  point  have  been  clipped  as  a  result  of  this  outlier  removal,  what 
remains  are  things  that  have  not  changed  dramatically. 
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Figure  37.  Cloud  Compare  software  created  figure  of  the  entire 
Pacific  Beach,  CA,  point  cloud  comparison  test  area 
showing  the  results  of  the  “first  cut”  removal  of 
difference  outliers  (85%  unchanged,  15%  change  outlier). 
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Focusing  on  the  difference  data  that  did  not  change  with  the  San  Diego  LiDAR 
and  WV-1  stereo  imagery  presented  an  unexpected  surprise.  The  outlier  and  90% 
confidence  interval  clipping  process  detected  an  area  of  large  scale  change  that 
corresponded  to  new  home  construction  during  the  time  of  the  LiDAR  collection  (2005) 
and  the  WV-1  stereo  image  collection  (2008)  (Figure  37).  The  area  that  has  changed,  in 
the  upper  right  comer,  stands  out  very  noticeably  as  rectangular  blue  areas  (Figure  37,  red 
box).  This  is  seen  more  clearly  in  Figure  38.  It  is  easy  to  pick  out  the  7  houses  in  the 
upper  right  comer  of  Figure  37  and  central  part  of  Figure  38  that  have  been  built  between 
the  2005  to  2008  time  frame  (normal  -  time  variant  -  change).  Notice  that  even  with  the 
predominant  noise  around  the  homes  and  in  the  tree  areas,  the  real  change  is  easily 
noticeable  as  large  “holes”  in  the  data.  It  is  also  easy  to  pick  out  the  taller  building 
structures  because  of  the  auto-correlation  outliers  along  the  building  edges  that  have  been 
removed. 
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Figure  38.  A  closer  inspection  of  the  new  home  construction  showing  the 
results  of  the  “first  cut”  outlier  removal.  The  “holes”  in  the 
data  show  where  new  homes  were  built. 


The  detection  of  this  new  home  construction  was  unexpected  and  did  show  that 
the  outlier  removal  process  does  work  to  detect  large  scale  change.  The  key  is  to  focus 
on  the  things  that  stayed  the  same,  clip  out  the  noise,  and  what  you  are  left  with  is  a  data 
set  that  can  show  large  change  if  it  exists. 

E.  PRECISION  IMAGERY-LID AR  FUSED  PRODUCT 

The  next  logical  step  is  to  focus  on  selected  aspects  of  creating  a  fused  optical 
image  and  LiDAR  data  product  viable  for  extraction  of  precision  3D  coordinates.  This  is 
different  than  assigning  a  Red-Green-Blue  (RGB)  value  to  each  LiDAR  point  as  is 
commonly  done  with  LiDAR  visualization  tools  and  is  supported  by  the  standardized 
LiDAR  file  format  (.las).  This  concept  is  innovative  in  that  the  idea  is  to  preserve  the 
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perspective  view  of  the  image,  preserve  the  native  resolution  of  the  image,  and  use  the 
coincident  LiDAR  data  to  facilitate  precision  3D  coordinate  extraction  (Rodannel  et  ah, 
2006). 

The  best  way  to  approach  this  concept  is  to  first  build  a  stereo  image  base  layer 
that  has  been  registered  to  LiDAR.  The  reverse  of  this  can  be  done,  which  would  be 
registering  the  LiDAR  to  the  base  layer  if  the  stereo  image  base  layer  is  considered  to  be 
more  accurate.  In  either  case,  a  stereo  image  base  layer  and  LiDAR  data  set  should  exist 
and  should  be  registered  such  that  they  are  coincident.  Having  these  two  data  sets 
coincident  provide  for  many,  very  attractive  options. 

One  option  is  that  LiDAR  data  can  be  converted  to  a  high  resolution  DEM  and 
displayed,  as  individual  points,  in  the  stereo  imagery.  This  provides  measurable  quality 
assurance  for  the  LiDAR  data  as  well  as  the  stereo  image  models. 

A  second  option  is  that  if  the  stereo  imagery  has  a  stated  accuracy,  the  LiDAR 
data  accuracy  can  be  either  verified  or  computed.  For  automated  accuracy  determination, 
the  LiDAR  accuracy  can  be  computed  by  using  an  autocorrelated  stereo  image  point 
cloud  to  LiDAR  point  cloud  change  detection  process  outlined  in  this  thesis.  Even  with 
the  noise  of  the  stereo  autocorrelation,  a  statistical  change  can  be  calculated  allowing 
absolute  and  relative  accuracy  to  be  computed. 

In  the  most  likely  scenario,  a  new,  more  current  and  detailed  image  would  be 
collected  that  would  need  to  be  registered  to  the  LiDAR  data.  In  this  case,  the  stereo  base 
layer  provides  for  an  attractive  auto-registration  layer  that  could  be  used  to  adjust  the  new 
image.  Once  coincident  with  the  stereo  base  layer,  the  image  can  then  be  easily  fused 
with  the  LiDAR  data  as  it  too  is  coincident  with  stereo  base  layer. 
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V.  SUMMARY 


LiDAR  point  cloud  data,  imagery,  and  stereo  image  derived  point  cloud  data  are 
ideal  candidates  for  data  fusion  to  provide  high  precision  digital  elevation  models  and 
improve  change  detection.  In  order  for  accurate  fusion  to  occur,  however,  the  LiDAR 
and  imagery  must  be  coincident. 

This  research  demonstrates  the  importance  of  registering  optical  imagery  to 
LiDAR  data  as  a  basis  for  proper,  seamless  fusion.  Several  dissimilar  image  data  types 
were  chosen  to  test  optical  image  to  LiDAR  registration  methods,  including  an  airborne 
panoramic  film  image  (OBC),  high  resolution  stereo  frame  color  images  (UltraCam 
Eagle),  and  panchromatic  satellite  images  (WV1).  Likewise,  several  LiDAR  sets  of 
varying  point  densities  and  quality  were  chosen  including  LiDAR  of  Haiti  at  3.4 
points/nr,  the  NPS  Campus  at  30.0  points/m-,  and  a  portion  of  San  Diego,  California 
LiDAR  at  1.4  points/m2. 

Individual  points  in  a  LiDAR  point  cloud  are  not  easy  to  precisely  identify  on  an 
image,  consequently,  methods  for  optical  image  to  LiDAR  registration  were  researched 
with  the  realization  that  the  most  practical  method  is  to  use  the  LiDAR-derived  intensity 
image  as  an  X,Y  (Latitude,  Longitude)  control  base  and  the  LiDAR-derived  DEM  as  the 
Z  or  height  control  base.  A  logical  control  point  selection  method  was  established  that 
used  features  that  were  homogeneous  in  surrounding  height.  This  helped  with  extracting 
a  more  precise  height  at  a  given  X,Y  location  because  the  height  could  be  interpolated 
from  the  LiDAR  derived  DEM. 

A  new  and  robust  process  for  registering  optical  imagery  to  LiDAR  was 
developed  as  part  of  this  research.  This  new  method  uses  a  two-step  triangulation 
process  that  derives  ground  control  from  the  LiDAR  intensity  image  in  one  step  and  then 
adjusts  the  imagery  in  a  second  step.  The  advantages  of  this  new  method  are  that  all 
measured  points  are  fully  reviewable  and  the  initial  triangulation  process  provides  quality 
assurance  of  the  intensity  image  control  points. 
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These  new  methods  and  processes  were  applied  to  data  for  three  test  sites.  In  the 
first  test  scenario,  OBC  image  adjustment  to  LiDAR  for  an  area  in  Haiti  proved  to  be 
problematic  using  the  panoramic  math  model  as  the  image  and  control  points  did  not 
match  well  (RMS  1.9  meters).  A  better  approach  used  the  RPC  representation  of  the 
panoramic  math  model  followed  by  an  affine  transfonnation  to  fit  the  image  to  control 
points,  with  an  improved  RMS  of  0.5  meters.  The  well-registered  results  pennitted  high 
resolution  visualization  of  the  OBC  imagery  and  Haiti  LiDAR  point  cloud.  The  RPC 
adjustment  approach  should  help  to  streamline  future  OBC  image  and  LiDAR  fusion 
processing  and  bring  more  utility  to  the  OBC  imagery,  as  RPC  adjustments  are  supported 
by  more  software  packages  than  rigorous  panoramic  math  models.  The  OBC  imagery 
was  not  in  stereo  (a  single  image)  so  no  further  point  cloud  analysis  was  performed  with 
this  data  set. 

Fusion  of  high  resolution  optical  aerial  imagery  and  dense  LiDAR  data  of  a 
portion  of  Monterey,  California,  over  NPS  campus  was  very  straightforward.  This  well 
calibrated  imagery  had  very  good  positional  and  attitude  information,  resulting  in 
precision  stereo  pairs  that  needed  no  triangulation  to  remove  parallax  or  control  points  to 
register  the  imagery  to  the  LiDAR.  The  data  sets  were  virtually  coincident  from  the  start, 
resulting  in  easy  fusion  of  the  imagery  and  LiDAR  point  cloud.  The  high  resolution 
nature  of  the  imagery  and  LiDAR  data  allowed  for  excellent  3D  visualization  of  the  NPS 
campus. 

For  the  NPS  dataset,  a  high  resolution  DEM  was  automatically  collected  from  the 
stereo  images  and  this  DEM  was  converted  into  a  point  cloud.  Several  automatic  DEM 
extraction  strategies  were  evaluated  in  the  attempt  to  create  a  quality  DEM.  After  these 
strategy  tests,  it  was  realized  that  autocorrelation  errors  will  always  be  present  and  the 
perfect  stereo  collected  point  cloud  cannot  be  obtained  from  autocorrelation  alone. 

The  NPS  LiDAR  point  cloud  and  the  stereo  point  cloud  were  then  used  as 

separate  entities  to  create  a  change  detection  layer.  A  new  point  cloud  was  created  that 

contained  the  difference  between  the  LiDAR  point  cloud  and  the  stereo  point  cloud. 

Using  this  “difference  cloud”,  a  unique  strategy  was  devised  to  focus  on  things  that 

stayed  the  same  as  opposed  to  focusing  on  things  that  were  different.  This  strategy  used 
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a  90%  confidence  interval  bracketing  method  that  clipped  outliers  and  autocorrelation 
errors.  Surprisingly,  this  method  worked  well  to  show  that  buildings,  objects,  and  trees 
were  generally  unchanged  even  though  30%  of  the  points  were  noise  caused  by  stereo 
image  autocorrelation  errors. 

The  final  test  case  investigated  WV 1  stereo  image  registration  to  LiDAR  data  and 
stereo  image  derived  point  cloud  versus  LiDAR  point  cloud  change  detection  for  the  San 
Diego,  California,  site.  The  improved  registration  process  was  used  to  triangulate 
(remove  y-parallax)  and  register  (control)  three  WV1  images  to  the  LIDAR  intensity 
image  and  LiDAR  DEM  resulting  in  a  good  registration  solution  (RMS  0.6  meters). 

Following  registration  of  the  three  WV  1  images,  several  autocorrelation  strategies 
were  attempted  with  the  stereo  imagery  to  try  to  create  the  perfect  point  cloud.  The  best 
WV  1  autocorrelation  point  cloud,  however,  was  still  very  noisy.  A  difference  point  cloud 
was  created  using  the  90%  confidence  interval  bracketing  method  to  clip  outliers  and 
autocorrelation  errors,  resulting  in  detection  of  unforeseen  urban  change  in  the  fonn  of 
new  home  construction  in  the  image. 

A  short  description  of  the  concepts  for  creating  a  fused  LiDAR/Image  product 
was  presented  focusing  on  the  requirement  of  using  an  accurate  and  coincident  stereo 
image  base  layer  as  the  foundation.  Accuracy  can  be  computed  for  the  LiDAR  data  using 
the  same  stereo  image  point  cloud  to  LiDAR  point  cloud  change  detection  process 
outlined  in  this  thesis  regardless  of  the  high  potential  for  stereo  autocorrelation  errors.  If 
a  new,  more  current  and  higher  resolution  image  is  collected,  the  stereo  base  layer  can  be 
used  in  a  photogrammetric  auto-registration  routine  to  align  the  new  image  with  the 
stereo  imagery  with  the  added  result  of  being  coincident  with  the  LiDAR  data  for  easy 
fusion. 
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VI.  CONCLUSIONS 


This  research  focused  on  demonstrating  the  processes  and  techniques  of 
registering  photogrammetric  imagery  to  LiDAR  point  cloud  data,  and  the  importance  of 
this  registration  as  the  precursor  for  data  fusion.  Point  cloud  data  derived  from  optical 
stereo  images  were  compared  to  LiDAR  point  cloud  data  and  basic  change  analysis  was 
performed.  This  work  is  a  precursor  to  a  larger  initiative  to  create  a  “ready  to  use” 
photogrammetric  image  and  LiDAR  fused  data  set. 

Most  image  to  LiDAR  registration  processes  utilize  edge  matching  techniques, 
however,  this  requires  identifiable  object  edges  throughout  the  image  and  sophisticated 
software  to  identify  and  extract  those  edges.  The  LiDAR  intensity  image  approach 
demonstrated  here  proved  more  capable  as  it  did  not  require  the  image  to  contain  object 
edges  which  allowed  registration  of  many  image  types  with  varying  scene  content.  The 
intensity  image  approach  was  also  easily  adapted  from  known  image-to-image 
registration  approaches  and  a  new  procedure  was  developed  for  this  process.  A  unique 
individual  point  selection  technique  that  requires  the  point  to  be  in  a  homogeneous  height 
area  was  successful  because  it  minimized  the  amount  of  derived  LiDAR  DEM  height 
interpolation  error. 

A  new,  2-step  triangulation  process  coupled  with  an  RPC  math  model  approach 
was  used  to  improve  the  registration  of  the  OBC  panoramic  imagery  to  LiDAR  data. 
Significant  improvement  was  achieved,  from  an  RMS  of  1 .9m  using  the  rigorous  model 
to  an  RMS  of  0.5  meters  using  the  RPC  method.  The  RPC  adjustment  approach  should 
help  to  streamline  future  OBC  image  and  LiDAR  fusion  processing  and  bring  more 
utility  to  the  OBC  imagery,  as  RPC  adjustments  are  supported  by  more  software 
packages  than  rigorous  panoramic  math  models.  For  all  image  types,  the  2-step 
triangulation  had  the  added  benefits  of  retaining  registration  point  measurements  for 
quality  review  and  intermediate  control  point  derivation  output  for  quality  assurance. 

High  resolution  DEM  extraction  from  stereo  imagery  can  have  as  much  as  30% 
autocorrelation  failure  principally  near  building  edges  and  tree  canopy.  This  high  rate  of 
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error  was  overcome  when  performing  stereo  image  and  LiDAR  point  cloud  change 
analysis  by  clipping  the  outliers  by  a  90%  confidence  interval  bracketing  method  to 
create  a  new  point  cloud  by  differencing  the  LiDAR  point  cloud  and  the  stereo  image 
point  cloud.  The  display  and  analysis  of  the  difference  point  cloud  resulted  in 
measurable  change  detection.  This  technique  produced  results  that  detected  small  scale 
(<50  cm)  differences  and  unforeseen  large  scale  change  in  the  form  of  new  home 
construction. 

Finally,  a  concept  for  a  fused  image  and  LiDAR  data  set  was  presented  that 
focuses  on  creating  a  stereo  image  base  layer  that  is  coincident  with  the  LiDAR  data. 
This  concept  has  many  benefits  including  stereo  image  and  LiDAR  accuracy  verification, 
LiDAR  accuracy  computation,  and  easier  succeeding  image/LiDAR  registration  and 
fusion.  This  research  and  these  benefits  justified  the  proposal  for  a  fused  LiDAR  and 
stereo  base  layer  as  the  foundation  for  further  LiDAR/image  fusion. 
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